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THE CHARACTERIZATION OF THETA-DISTINGUISHED 
REPRESENTATIONS OF GL n 

EYAL KAPLAN 


Abstract. Let 9 and 9' be a pair of exceptional representations in the sense of Kazhdan 
and Patterson IKP8-1I . of a metaplectic double cover of GL„. The tensor 9 ® 9' is a (very 
large) representation of GL n . We characterize its irreducible generic quotients. In the 
square-integrable case, these are precisely the representations whose symmetric square L- 
function has a pole at s = 0. Our proof of this case involves a new globalization result. In 
the general case these are the representations induced from distinguished data or pairs of 
representations and their contragradients. The combinatorial analysis is based on a complete 
determination of the twisted Jacquet modules of 9. As a corollary, 9 is shown to admit a 
new “metaplectic Shalika model”. 


1. Introduction 

Let F be a local non-Archimedean field of characteristic zero. Let 9 and 9' be a pair of 
exceptional representations in the sense of Kazhdan and Patterson (KP84] , these are repre¬ 
sentations of a metaplectic double cover GL n = GL n (F) of GL n = GL n (E). The GL n -module 
9 ® 9' has appeared, in both local and global incarnations, in several studies, most notably 
in the work of Bump and Ginzburg [BG92 ] on the global symmetric square L-function. 
Locally it has been studied by Savin [Sav92j and Kable [KabOli Kab02], who considered 
multiplicity-one properties and spherical quotients. In a more general context, exceptional 
or minimal representations have played an important role in the theta correspondence, the 
descent method and Rankin-Selberg integrals [GRS031 IGin06l iGJSllj . 

Our main goal in this work is to characterize the generic quotients of 9® 9' . An admissible 
representation r of GL n is called distinguished if 

Hom GLn (0® 9',t v ) t 0, 

where r v is the representation contragradient to r. Here is our main result. 

Theorem A. Let r be an irreducible generic representation of GL n . 

(1) If t is essentially square-integrable, then it is distinguished if and only if it is unitary 
and the symmetric square L-function L(s,r, Sym 2 ) has a pole at s - 0. 

(2) In general write r as a parabolically induced representation r - q x ... x r m , where 
each Ti is essentially square-integrable. Then r is distinguished if and only if there is 
0 < m 0 < [m/2j such that, perhaps after permuting the indices of the inducing data, 
r 2 i = Tfi_i for 1 < i < m 0 and r* is distinguished for 2m 0 + 1 < i < m. 

The supercuspidal case has already been proved in | Kapl4c| . The square-integrable case 
is handled in Theorem 14.121 It relies on a new globalization result (see below). In this case 
there is one assumption, caused by the globalization method: when t is square-integrable 
with a nontrivial central character and L(s,r, Sym 2 ) has a pole at s - 0, the proof that r is 
distinguished relies on the existence of the local functorial lift of Cogdell et al. |CKPSS04j 
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for quasi-split special even orthogonal groups (the global lift was described in |CPSSllj ). 
See Remark 14.Ill for more details. 

The fact that parabolic induction (irreducible or not) of distinguished representations is 
also distinguished, i.e., an upper heredity result, was established in |Kapl4b|. In Theorem 14.41 
we show that for an irreducible r (generic or not), txt v is always distinguished. The structure 
of generic distinguished representations in general is given in Theorem 14.161 

Now consider, for example, the representation v l ! 2 p x z/ -1 / 2 p, where v = | det | and p is 
an irreducible unitary self-dual supercuspidal representation of GL ra . It is of length 2, has 
a unique irreducible quotient Lang (v 1 ! 2 p x v~ l ! 2 p) - the Langlands quotient, and a unique 
irreducible subrepresentation A, which is square-integrable ( |Zel80j 9.1). Theorem 14.41 implies 
that v l ! 2 p x ir x l 2 p is distinguished. Then Theorem lAl shows that, depending on the pole of 
L(s, A,Sym 2 ) at s - 0, A is distinguished or not. But in the absence of a pole, we deduce 
that Lang(iR / 2 /9 x z/ -1 / 2 p) is distinguished. This is an example of a non-generic distinguished 
representation. 

Another immediate corollary of Theorem [A] is that an irreducible generic distinguished 
representation must be self-dual. 

Savin |Sav92] considered the spherical quotients of 9 ® 9'. He conjectured that a spherical 
representation with a trivial central character is distinguished, if and only if it is the lift 
of a representation of a prescribed classical group. The case of n - 3 was established in 
|Sav92] . Kable |Kab02] proved that such lifts are distinguished, for a general n, using 
analytic methods. The other direction was settled in |Kapl4b|, by extending the ideas of 

[B5552] . 

In more detail, let B n = T n tx N n be the Borel subgroup of GL n , where N n is the subgroup of 
upper triangular unipotent matrices, and Q n -k,k = M n -k,k x U n -k,k be the standard maximal 
parabolic subgroup with M n - k ,k = GL n - k x GL/,. In |K apl4b| we described a filtration of 
as a T n - module, using the theory of derivatives of Bernstein and Zelevinsky [BZ76'. 
1BZ77] . The techniques of |Kapl4b| break down in the general setting, where we have a 
Jacquet module of 9®9' with respect to a unipotent radical of an arbitrary standard parabolic 
subgroup. The theory of derivatives is no longer applicable, the main barrier being that the 
action of M n _ kk on the set of nontrivial characters of U n _ k ,k is not transitive. 

In fact, the arguments here can also be used to deduce the structure of distinguished 
spherical representations, but the claims in |Kap l4b] are actually stronger: the appearance 
of the inducing character as a subquotient of (9®9 r ) Nn determines its combinatorial structure 
completely. 

In this work we take a different approach, and rely on the computation of all the twisted 
Jacquet modules of 9 , corresponding to maximal parabolic subgroups. Let 0 < k < n and 
fix an additive character ^ of F. The action of M n _ k ,k on the characters of U n - k ,k has 
min(n -k,k) + 1 orbits, we fix representatives ipj, where 0 < j < min(n - k, k ). The character 
V’o is the trivial character, and when n = 2 k, is the generic character in the sense that 
its stabilizer Gh k in M k k is reductive. The group Gh k is simply the diagonal embedding of 
GL*. in M k)k . Restriction of the cover of GL 2 fc to Gh k is a trivial double cover (GL/. splits 
under this cover). 


Theorem B. For any 0 < j < min(n - k,k), the Jacquet module of 9 with respect to U n - k: k 
and fjj is isomorphic to {9 n -k-j®9 k -j)® r )^(_iy-i'if. Here 9 m is an exceptional representation 
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of GL m , ® is the metaplectic tensor of Kable jKabOlj . is a one-dimensional rep¬ 

resentation of Ghj ix Ujj, and T^-iy-i is a certain Weil factor regarded as a character of 

GIL,-. 


The main ingredient in the proof is the case of n - 2k = 2j. When n - 2, ifi is the usual 
Whittaker character of A r 2 and this result was proved by Gelbart and Piatetski-Shapiro 
[ GPS80 ] (Theorem 2 . 2 ). The general case is proved in Theorem 13.II using induction and the 
local “exchange of roots” of Ginzburg, Rallis and Soudry [ GRS99 ]. For the global analog, 
one may utilize the global “exchanging roots” ( |Gin901 [CmSCTIl ISou05[ lGRSTT] h The result 
for an arbitrary maximal parabolic is given in Theorem 13.31 

Kable jKabOlj computed the nontwisted Jacquet modules, i.e., with the trivial character 
'0o, and expressed them as metaplectic tensors. This already had several applications iKabOll 
|Kab02| [Kapl4b| . 

The particular case of n - 2k and the generic character 0*. implies a notion of a “meta¬ 
plectic Shalika model” (0 fc takes the form of the usual Shalika character). In this case, 
Theorem [B] implies the existence of a unique (up to a scalar) nontrivial linear functional on 
the space of 9, which is ^-equivariant on the left under Uk,k■ In the non-metaplectic setting, 
for the existence of a Shalika functional, the stabilizer GIL*, must then act trivially. The 
correspondence between representations of GL^ and GL^ (recall that this cover is trivial) 
takes 1 to 1 ® 7 ^, which is essentially the representation 7 ^ we obtained. I 11 turn, we 

deduce a metaplectic Shalika model for 9 , which we use for the proof of Theorem 14.41 For 
more details see Remark 13.21 We mention that this model may have applications similar to 
those of [ FJ93 . JR96] . 

Let GSpin 2n+1 be the split odd general spin group of rank n + 1. A result similar to 
Theorem [B] for exceptional representations 0 of GSpin 2ra+1 (defined in | Kapl4a| following 
the exposition in (BFG03 ] for S0 2 n+ i), was proved in | Kapl4c| and used in a study of 0®0'. 
The definition of distinguished representations of GSpin 2n+1 is similar to that of GL n , an 
irreducible representation is distinguished if its contragradient is a quotient of 0 ® 0'. The 
results of |Kapl4c| allow us to alternate between quotients of 9®9' and 0®0', using parabolic 
induction. 

One essential difference between these settings, is that in contrast with 9 ® 9 ', the space 
0 ® 0' does not afford a Whittaker functional ([Kapl4]c] Theorem 1). This interplay will 
be used as an ingredient in a forthcoming work on the conjecture of Lapid and Mao on 
Whittaker-Fourier coefficients, for even orthogonal groups ( |LM13j ). One can also use this 
to deduce certain irreducibility results (see R,emark 14.1511 . I 11 more detail, Proposition 4.1 of 
|Kapl4c| showed that for a tempered distinguished r, a representation parabolically induced 
from z/ 1 / 2 r® 1 to GSpin 2n+1 will also be distinguished, but then it must be reducible, otherwise 
it is a generic quotient of 0 ® 0 '. 

A result similar to a global formulation of Theorem |B] was used to compute a global co¬ 
period integral, involving the integration of the residue of an Eisenstein series, against a 
pair of automorphic forms in the spaces of the small representation of S0 2n +i or GSpin 2n+1 
|Kap| |Kapl4a| . 

To prove that a square-integrable representation r such that L(s,r, Sym 2 ) has a pole at 
s - 0 , is distinguished, we apply the following globalization lemma. As mentioned above, in 
one case its proof relies on an assumption, explicitly given in Remark 14. Ill 





























































4 


EYAL KAPLAN 


Theorem C. Let tt be a square-integrable representation of GL n . Assume that L(s,7t,R) 
has a pole at s - 0, for R = Sym 2 or A 2 - Then there exist a number field with a ring of adeles 
A and a global cuspidal representation II of GL n (A), such that for a sufficiently large finite 
set of places S, L 5 (s,n,R) has a pole at s - 1, and at some place v, IR = n. 


See Lemma 14.91 This result is expected to have other applications. 

There is a minor “historical gap” in the theory of exceptional representations, regarding 
their Whittaker models. Let F be any local non-Archimedean field of characteristic different 
from 2. Kazhdan and Patterson [KP84J proved that for n > 3, if |2| ± 1 in F, the exceptional 
representations do not have Whittaker models. For n - 3, Flicker, Kazhdan and Savin 
[FKS90] (Lemma 6) used global methods to extend this result to the case |2| = 1. The 
remaining case (|2| = 1 and n > 3) has been expected, but not proved. We complete the 
proof in Theorem 12.61 This immediately validates several results from [BG921 [KabOlj lKab02l 
Kapl4b| when the held is dyadic, including the local functional equation of [BG92] (Section 5) 


and the aforementioned conjecture of Savin (see IKab02] p. 1602), which now hold in general. 

Exceptional representations for GL n were introduced and studied by Kazhdan and Pat¬ 
terson [KP84 ], They are related to a broader class of small, or minimal, representations. 
The first example was probably the Weil representation of Sp n . These representations enjoy 
the vanishing of a large class of Fourier coefficients, which makes them valuable for applica¬ 
tions involving lifts and Rankin-Selberg integrals [ GRS03 , iGin06l iGJSllj . They have had a 
profound role in the theory of representations and appeared in numerous studies, including 
Vog8l| IKaz901 IKS901 EM51IBKM ISav941 IGR.S971 iBFGOOl iGRSnll IKPW921IBFG031 USHSl 


iGSUhl ISILS081 IGRSllj . 


To date, the significant application of the exceptional representations of Kazhdan and 
Patterson [ KP84j . is the construction of a Rankin-Selberg integral representation for the 
symmetric square L- function, by Bump and Ginzburg [ BG92j . Takeda (Takl4j extended the 
results of I BG92 .. to some extent, to the twisted symmetric square L-function. 

The term “distinguished” has been used in various contexts. Let £ be a representation 
of a group G and let rj be a character of a subgroup H < G. The representation £ is 
called (H, ^-distinguished, if Hom^(£,77) + 0. There are numerous studies on local and 
global distinguished representations, including |Jac91l [JR921 IFJ93l IQff061 IQS071IQS081 IQff09l 
IJaclOl IMatlOal IMatlObl IMatlll IFLQ121 IMatj . 

Let F 0 be a quadratic extension of F. Matringe jMatlOallMatlObUMatllj studied (GL n (F), r/)- 
distinguished representations of GL n (F 0 ). He proved, using different (but in some sense 
related) techniques, a combinatorial characterization similar to Theorem [A] He also proved 
( [Mat 10b] ) that an irreducible generic representation £ is distinguished, if and only if its 
Rankin-Selberg Asai L-function has an exceptional pole at 0. He then showed that L(s, £, Asai) = 
L(s,p(£), Asai), where p(£) is the Langlands parameter associated with £ QMat09| h 

Feigon, Lapid and Offen jFLQ12] studied (local and global) representations distinguished 
by unitary groups. 

The rest of this work is organized as follows. Section [2]contains the preliminaries, including 
a brief review of the metaplectic tensor of Kable jKabOlj and exceptional representations. 

The Jacquet modules are computed in Sectional Our results on distinguished representations 
occupy Section m 
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Soudry for communicating the results of Liu [Liull] and Jantzen and Liu j JL14J to me. 1 
thank Baiying Liu for helpful conversations. I would like to thank Chris Jantzen for his help 
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2. Preliminaries 

2.1. The groups. Let F be a local non-Archimedean held of characteristic different from 
2. Let (,)2 be the Hilbert symbol of order 2 of F and put p 2 = {-1,1}. For a group G , 
Cg denotes its center. If x,y e G and Y < G, x y - xyx -1 , X Y = { x y : y e Y}. Also if d e Z, 
Y d = {y d : y e Y} and in particular F* d = ( F *) d . The set of m x k matrices over F is denoted 

pmxk 

Note that our results up to and including Corollary 14.71 apply to any local p-adic held of 
characteristic different from 2. Then we add the assumption that the characteristic of F is 
0 (see also Remark l4.18p . Henceforth we omit references to the held, e.g., GL n = GL„(F). 

Fix the Borel subgroup of upper triangular invertible matrices B n = T n tx N n , where T n is the 
diagonal torus. A partition of n into m parts is an m-tuple a = (cq,..., a m ) of nonnegative 
integers, whose sum is n (partitions will be regarded as ordered partitions). For a partition 
a, let Q a = M a x U a denote the corresponding standard parabolic subgroup; its Levi part 
M a is isomorphic to GL ai x ... x GL am . In particular the maximal parabolic subgroups are 
the subgroups Q n -k,k for 0 < k < n; Q n - Qo,n = Qn, 0 = GL n . The group GL^ is regarded as a 
subgroup of GL n via its natural embedding in M k , n -k- For any parabolic subgroup Q < GL n , 
let 5q denote its modulus character. 

Let GL„ be the metaplectic double cover of GL„, as constructed by Kazhdan and Patterson 
[KP84] (with their c parameter equal to 0). They defined their cover by restriction from the 
cover of SL n+1 of Matsumoto [ Mat69 ]. We use the block-compatible cocycle a - a n of Banks, 
Levi and Sepanski [BLS99j . which coincides with the cocycle of Kubota [ Kub67j for n = 2. 
The block-compatibility property reads, for a, a' e GL^.fc and b,b' e GL^, 

(2.1) a(diag(a, b), diag(a', b')) = (7 n _*,(a, a')cr k (b, 6')(det a, det b') 2 . 

Let s : GL n -»■ GL n be the section of [ BLS99 ] and p ■ GL n -»■ GL n be the natural projection. 
For any subset X c GL n (F), denote X = p^(X). We pull back the determinant to a non- 
genuine character of GL n , also denoted det, and using this any character of F* can be pulled 
back to a character of GL n . Let e be 1 if n is odd, otherwise e = 2. Then = P -1 (C , gl„)- 

2.2. Representations. Let G be an /-group ([ BZ76 ] 1.1). Representations of G will be 
complex and smooth. Let Alg G denote the category of these representations and Alg irr G c 
AlgG* the subcategory of irreducible representations. For n e Alg G, 7r v is the representation 
contragradient to n. If 7 r admits a central character, it will be denoted uv If Ft < G and 
g € G, 9 7t is the representation of 9 FI on the space of 7r given by 9 tt(x ) = 7r( 9 x ). We say 
that 7r is glued from representations 7Ti,... , 7 r^, if 7r has a filtration, whose quotients (which 
may be isomorphic or zero) are, after a permutation, ni,... 

Assume that G is a given central extension of G by ^2 and ip ■ G -»■ G is a section. If 7 r 
and 7 r' are genuine representations of G, n ® 7r' (outer tensor product) is a representation 
of G via g ^ 7r (<p(g)) 7r '(p(g)). The dehnition is independent of the choice of tp and the 
actual section will be omitted. 

Let X be an /-space ( |BZ76j 1.1). The space of Schwartz-Bruhat functions on X is denoted 
S(X). If G acts on X, it also acts on <S(W). 
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Regular induction is denoted Ind and ind is the compact induction. Induction is not 
normalized. In GL n , if a = (a 1; ..., a m ) is a partition of n and T\ ® ® r m e Alg M a , i.e., 

Ti e AlgGL Q ., then r, x ... x r m = Indg|' n (A^r, ® ® r m ). 

Let Alg sqr GL n c Alg csqr GL n c Alg irr GL n be the subcategories of square-integrable and 
essentially square-integrable representations. We briefly recall the characterization of essen¬ 
tially square-integrable representations of Zelevinsky |Zel80] (Section 9). Let Alg cusp GL n c 
Alg esqr GL n denote the subcategory of supercuspidal representations (not necessarily uni¬ 
tary). For brevity, put v = | det |. Let ^ be the set of equivalence classes of Alg cusp GL n for 
all n > 0. A segment in ^ is a subset \p. iA _1 p] - {p, up ..., zA -1 p}, where p e Alg cusp GL^ and 
l > 1 is an integer. The representation pxupx.. .x iA _1 p has a unique irreducible quotient de¬ 
noted ([p, i/ z_1 p])L Then r € Alg esqr GL n if and only if it is isomorphic to some representation 
([p, z/ -1 p])L In this case u T = z/O -1 )/ 2 ^. Also note that (([p, v l ~ 1 p]) t )' / = ([u 1 ~ l p v , p v ])L 

Zelevinsky |Zel80 j also described the reducibility of products of essentially square-integrable 
representations. Two segments [p, iA _1 p] and [p', v v ~ x p'\ are called linked, if neither of them 
is a subset of the other, but their union is a segment. If they are linked, in particular p = v a p' 
for some integer a. The representation ([pi, ^ 1_1 pi])* x ... x ([p m , P im_1 p m ]) 4 is irreducible if 
and only if no pair of segments are linked. 

Let ip he a nontrivial additive character of F. Denote the normalized Weil factor by 7 ^, 
( |Wei64j Section 14), 7 y (-) 4 = 1. For a e F*, let 7 <y a be the Weil factor corresponding to the 
character x >-+ ax ). Recall the following formulas (see the appendix of [Rao93j ): 

(2.2) ^(xy) ='yi > (x)'yi > (y)(x,y) 2 , 7y(^ 2 ) = 1, 1^A X ) = (a,x) 2 ^(x). 

We also denote by ip the generic character of N n given by ip(u) = iP(Ya=i u i,i+ 1 )- By restric¬ 
tion, it is a character of any unipotent subgroup U < N n . When we define a character of 
such a subgroup, we usually use the notation ip, if it coincides with this restriction. 

Let r € Alg irr GL n . A ( , 0-)Whittaker functional on r is a functional A such that A(it<p) = 
ip{u)\(ip) for any u e N n and <p in the space of r. We say that r is generic, if it admits a 
nontrivial Whittaker functional. 

If n = 2k we can also consider a Shalika functional, the equivariance property now reads 
M( C c)a) = , 0(t r (' y ))^( < p) f° r any c € GL fc and v € F kxk . If A + 0, it yields an embedding 
t c IndgL*[/(V0) where GL fc = {diag(c, c) : c € GL fc }, called a Shalika model of r. If it exists, 
uniqueness of this model was proved by Jacquet and Rallis [JR96 ] in the non-Archimedean 
case and by Ash and Ginburg [ AG94] over Archimedean fields. 

2.3. Jacquet modules. Let 7 r e AlgG. Let U < G be a closed subgroup, exhausted by its 
compact subgroups (here U will be a unipotent subgroup of GL n ) and ip be a character of 
U. Assume M < G is the normalizer of U and stabilizer of ip. The Jacquet module of n with 
respect to U and ip is denoted TTu,ip- The action is not normalized. We have the following 
exact sequence in Alg M, 

0 -> 7 r(U,ip) -» 7r -► 7 x Uti , -> 0. 

The representation 7 r(U,ip) can be characterized by the Jacquet-Langlands lemma: 

Lemma 2.1. (see e.g. |BZ76] 2.33) a vector v in the space of n belongs to 7r(U,il’) if and 
only if 

7 r(u)v ^ _1 (u) du = 0, 
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for some compact subgroup U <U. 

When if - 1, we simply write rc(U) and 7py. The following consequence of this lemma will 
be used. 


Lemma 2.2. Let Q be an l-group and U < Q be a closed subgroup, exhausted by its compact 
subgroups, which is normal in Q. The group Q acts on the set of characters ip of U by 
ip 9 (u) = V ; ( 9 u), denote an orbit by 0(ip). Assume ip and ip' are two characters of U and 
let L,L' < Q be their stabilizers. Assume L, L ', LU, L'U are closed subgroups of Q. Then for 
7 r € Alg LU and tt' e Alg L'U, 


( ind lu n UT ® ind %u >)u 


| ind^ (ttc/,-0 <2> tt^! ) ip’ = ip 1 , 

[() ’ fi'tO{iP~ l ). 


Proof of LemmalM Let / belong to the space of ind^ 7iu,ip- Choose a small compact open 
neighborhood of the identity V < Q and a finite set Q o Q, such that / is fixed by V and the 
support of / is contained in LULIV. Similarly let f belong to the space of ind^,^r^, and 
choose V' and f2', analogously. For any compact U <U and g,g' e Q , 

(2.3) f u-(f ® f)(g,g') du = f(g)® f'(g') f ip( 9 u)ip'( 9 'u) du. 

JZA JIA 

This clearly vanishes unless g = luxv , where leL, ueU,x^Ll and we V, and g' = I'u'x'v' 
(with a similar notation). In this case the integral equals 

(2.4) ip(u)7T UA ,(l)f(x) ®ip\u)'K , u ^{l')f\x') f ip( xv u)ip'( x ' v ’u) du. 

JIA 

Assume ip' { O^ip* 1 )- Then for any y,y' e Q, u ip( y u)ip'( y 'u) is a nontrivial character 
of U. Since x and x' vary in finite sets, while v and v' vary in compact neighborhoods of 
the identity, one can choose a large enough compact U such that the integral vanishes for all 
data x,v,x',v'. Thus Lemma [All implies that / ® f vanishes in the Jacquet module. 

Now assume ip' = ip~ l . Consider the mapping 

/ ® /' ^ (/ ® /')(#) = /(#) ® fig)- 

It is clearly onto ® i T 'u^-i)- If f ® f = 0 in the Jacquet module, by lemma 12.11 the 

left-hand side of (12.3j) vanishes for all g = g' e Q, but then the du-integral on the right-hand 
side becomes a nonzero constant. This implies (/ ® f')(g) = f(g) ® f'(g ) = 0. 

In the other direction assume f(g) ® f'(g) = 0 for all g e Q. We must show that (12.31) 
vanishes for all g, g' e Q. It clearly does if g - g'. Take g + g'. Using the notation above write 
g = luxv and g' = I'u'x'v' and consider the right-hand side of (12.31) . 

If the character u ^ ip^u^ip^tu'u) is trivial on U, g'g -1 = l"u" e LU, but then 

f(9 ) ® f'W) - f{g)®r 1 {u"y u ^i")f'{g). 

By our assumption either f(g) = 0 or f'(g) = 0 and in the latter case, 7T^_i (l")f'(g) = 0 for 
any l" e L. Thus f(g) ® f'(g') = 0. 

For a pair g,g' such that this character is a nontrivial character, as above using the fact 
that x, x' vary in finite sets and v,v' belong to compact subgroups, one can choose a large 
enough U so that this integral vanishes. 

Finally to pass from pure tensors to the general case, note that a general element of 
ind 2c/ TTyy ® ind'Uy tt' 7 ^ can be written as a finite sum fi®f' with the following property: 
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for any subset of indices q ,... ,i r such that the supports of f tj ® f[. coincide for 1 < j < r, 
for each g,g' in the support of f h ® /£, the vectors (/ n ® ■ ■ ■, (fi r ® f' r )(g,g') are 

linearly independent in 7r/y,q ® Then (X!,; ./) 0 f')(g,g') - 0 if and only if for each z, 

Ui®fl){g,g') = o. ’ □ 


2.4. Metaplectic tensor. Let a be a partition of n. Irreducible representations of M a can 
be described using the tensor product. The situation in M a is more complicated, because 
the direct factors of M a do not commute in the cover, then the tensor construction breaks 
down. An alternative definition of a metaplectic tensor was developed by Kable [KabOlj . we 
briefly describe his construction. Other studies of metaplectic tensor include [FK86. Sun97i 
IMez04i lTak!3j . The definition of Kable has several advantages, including the variety of 
useful properties it enjoys and its relation to the exceptional representations (see (12.71) and 
(EFHj) below). 

For any closed subgroup H < GL n , Let H a - {h e H : det h e F* 2 }. If n e Alg H, denote its 
restriction to H a by 7 t d . 

Let cq be a partition of n t and 7q e Alg irr M a . be genuine, i - 1,2. Put n - ni + n 2 and let 
a = (ai,a 2 ), i.e., a is the concatenation of a\ and a 2 - Then 

are pairs of commuting subgroups (see (12.ip ). The representation of p _1 (M° l x M°) 

is genuine and similarly, 7r° ® n 2 and 7Ti ® 7r°. Let w be a genuine character of which 
agrees with 


^1 I P- 1 {C 2 gLni ) 0 ^2 |p-l (C2 Ln2 ) 

on P _ 1 (C'gl„)- The metaplectic tensor 'K\® LO n 2 was dehned by Kable [KabOl] as an irreducible 
summand of 


(2.5) 


ind 


M a 

p- 1 (MS 1 xAfJj) 




on which Cgj-^ acts by oj. Note that the definition in [KabOlj was more general and included 
genuine admissible finite length representations, which admit a central character. Here we 
will only encounter the case of irreducible representations. 

A more concrete description was given in [KabOlj (Corollary 3.1): if n 2 is even or rii and 
n 2 are odd, there is an irreducible summand a c tt° such that 


(2.6) 2 = md^? (Mai><M p 2) (7r 1 ®> a). 

If both n 2 and ri\ are even, cr is arbitrary; if n 2 is even and ri\ is odd, a is uniquely determined 
by the condition u = (u 7ri ®u fT on Cqi ; when both are odd, a = i r°. Otherwise n 2 is odd and 
ri[ is even, the definition is similar with the roles of rq and n 2 reversed. 

The metaplectic tensor is associative ( [KabOlj Proposition 3.5) and if i3 t is a subpartition 
of oij, i.e., & is a partition of n, with Qp. < Q a ., 


(2.7) (vri®a jTr 2 )u Pl xUp 2 - 

( [KabOlj Proposition 4.1). 

Kable did not consider the behavior of the metaplectic tensor with respect to twisted 
Jacquet functors, i.e., with a nontrivial character z/;. Such modules are not representations 
of standard Levi factors. One may attempt to extend the definition, to some extent, to 
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include these cases. For our purposes it will suffice to use restriction, in order to compute 
twisted Jacquet functors applied to the metaplectic tensor. 


Claim 2.3. Let a^ be a partition ofrii, 7q e Alg irr M a . 

C r° ® 7T 2 

® _®_ 

gzMS 2 \M a2 


(7r 1 ® a ,7r 2 )|p-i(M° lX M Q2 ) - 


be genuine, i = 1,2. Then 
even n 2 , 

Xg ' K 2 odd n i and n 2 , 


© % ® X 3 7t 2 even rij, odd n 2 . 


Here Xg is the character of M°.\M ai given by Xg( x ) = (det x, det g) 2 and a is an irreducible 
summand of Similar results hold mutatis mutandis when restricting to p~ 1 (M ai x M° ). 
Consequently 


(7Ti® a; 7r 2 )p-l( M n iXM n 2 ) 


J[F* : F* 2 ]7t° ® 7r° ni and n 2 are odd, 
[ 77 ° ® 7r° otherwise. 


Proof of Claim \2.S\ The assertions follow from by Mackey’s theory, using (KabOlj 

(Propositions 3.1 and 3.2) and (12.11) . For details see |Kapl4b| . Note that the restriction 
to x ) was already computed in [KabOlj (Theorem 3.1). □ 


This claim has the following consequence, which will be used repeatedly. 

Corollary 2.4. Let a^ be a partition of ni, tt % , 7r' e Alg irr M Qi be genuine for i - 1,2, and 
a = (ai,a 2 ). Assume that for an admissible representation T\ ® t 2 <e Alg M a , 

Hom Ma ((7r 1 ®o;7r 2 ) ® {’ k [® u -' k 2 ), t 1 ® t 2 ) V 0. 

Then for each i there is some square-trivial character Xi of F*, such that 

Hom Ma . (Ti ® 7T-, Xip) * 0. 

Moreover, if ni is even, Xi is i n foot trivial. 


Proof of Corollary \2-4\ According to Claim 1231 and with the same notation, ('Ki® u} 'k 2 )\ p -i( M l c 
is a finite direct sum of representations 9 a ® Xg n 2 - Note that this form includes all the pos¬ 
sibilities listed in the claim (e.g., a could be 7r°), and if n 2 is even, Xg = 1- We use a similar 
notation 9 'a' ® Xg ,7r 2 f° r this restriction of Ti'f®^^. Hence for some g and g', 

Hom Mgi xMaa ((V ® 9 'a') ® (x 9 tt 2 ® Xg'^ 2 ), n ® r 2 ) * 0. 


J xMo 2 ) 


Thus if X 2 ~ XgXg'i Hoixl M q2 (7t 2 ® 7T2, y 2 r 2 ) ^ 0 as claimed (here we used the admissibility of 
Ti and r 2 ). If n 2 is even, Xg = Xg' = 1 hence y 2 = 1. The other case is symmetric. □ 


2.5. Exceptional representations. The exceptional representations were introduced and 
studied by Kazhdan and Patterson |KP84] . Let £ be an exceptional character, that is, £ is 
a genuine character of Cf = p ' 1 (Tff)C 'qi , such that 

£(s(diag(/;_i,x 2 ,aT 2 , /n-j-i))) = |x|, VI < i < n - l,x e F*. 

Let p(£) be the corresponding genuine irreducible representation of T n (for n > 1, T n is a 2- 
step nilpotent subgroup). The principal series representation Ind~ Ln (^ 2 p(x)) has a unique 
irreducible quotient 6, called an exceptional representation. 
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The exceptional characters £ are parameterized in the following manner. Let x be a 
character of F*. Let 7 : F* -> C* be a mapping such that 7 (xy) = 7(01)7(y)(x, y)!^^ and 
7(x 2 ) = 1 for all x,i/ e F*. We call such a mapping a pseudo-character. Note that the 
definition of a pseudo-character depends on n, to explicate this we will occasionally call it 
an n-pseudo-character. Define 

4 i7 (es(z/„)s(t)) = ej(z)x(z n e e /z 2 , t e T 2 , z e F* e . 

When n is even, the choice of 7 is irrelevant. When n = 1 ( 4 ), 7 is a square-trivial character 
of F*. If n = 3 ( 4 ), 7 = 7,/, for some nontrivial additive character ^ of F. Note that 
a(zl n ,z'l n ) = (z, z')^ 2 ^. The corresponding exceptional representation will be denoted 
0 n , x , 7 , or 9 xn when n is clear from the context. Furthermore, sometimes we simply denote 
an exceptional representation by 9 n or 9, when the data x and 7 do not affect the validity 
of the argument. Note that u>e x/y (s(zl n )) = x( z ) n T( z )- 

The following simple claim was proved in [Kap l 4 c| (in the proof of Claim 4 . 1 ): 

Claim 2.5. We have 6 xn = Additionally, if 70 is another pseudo-character, 9 xn - 

r]9 Xil0 for some square-trivial character rj of F*. 

Proof of Claim \275 1 The proof follows easily from the fact that 7/70 is a square-trivial char¬ 
acter of F*. □ 


The Jacquet functor carries exceptional representations into exceptional representations 
of Levi subgroups. The following result is due to Kable |Kab01j (Theorem 5.1), 


( 2 . 8 ) 


(« 


ni+n 2 ,x,'yJUn 


= <f 1/4 

0 Qn, 


&v-.9r, 


'ni,X.7WX7 l '«2,X,72 


Here 71 and 72 are arbitrary (in jKabOlj this appears with x - 1 an d with the normal¬ 
ized Jacquet functor). On the right-hand side 77 is regarded as the character es(zl n ) 

e x( z ) n x( z )- 

Kazhdan and Patterson [ KP84] (Section 1.3, see also |BG92j p. 145 and jKabOlj Theo¬ 
rem 5.4) proved that for n > 3, if |2| + 1 in F, the exceptional representations do not have 
Whittaker models. For n = 3, Flicker, Kazhdan and Savin |FKS90] (Lemma 6 ) used global 
methods to extend this result to the case |2| = 1. The following theorem completes the proof 
that 9 is non-generic also when |2| = 1 and n > 3. 


Theorem 2.6. For any n > 3, 9 - 9 n does not have a Whittaker model. 


Proof of Theorem \2. (A According to (12. 8 p (proved in [KabOlj without any assumption on the 
field), 9 c Ind~ Ln (j )/ 4 # 3 ®0 n _ 3 ). Hence it is enough to prove 

0; 3,n—3 ^3 ,n 3 




fl/4 


0 . 


By virtue of the Geometric Lemma of Bernstein and Zelevinsky ([ BZ7 TJ Theorem 5.2), this 
representation is glued from certain Jacquet modules of 0 3 ®0 n _ 3 . We show that these Jacquet 
modules vanish, this will complete the proof. Let W be a subset of Weyl elements such that 
GL n = IIwevv; Qz,n-zW l N n . When 'f\™u 3jl - :i nN, n + 1, the quotient corresponding to w vanishes. 
This implies there is only one quotient, corresponding to vj 0 = ( j In ~ 3 ), namely 


( 2 . 9 ) 


S- (0 3 ®^ri-3)A3xA r n _3,y- 
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Here 6 is some modulus character and N :i x N n _ 3 < M 3n _ 3 . According to Claim [2731 when we 
restrict 9 3 ®9 n _ 3 to GL 3 x GL°_ 3 , we obtain a finite direct sum of representations 9° ® #°_ 3 . 
Since the Jacquet functor with respect to N 3 and 0 commutes with this tensor and the 
restriction, and (0 3 )n 3 ,^ - 0 by [ FKS90 ] (Lemma 6), (12.9ft is zero. □ 

Lemma 2.7. Assume n > 3, 9 - 9 n and let 0 be the character of given by 0(ti) = 

■ 0 ( 141 , 2 ). Then 9uis a quotient of 9jj 2 2 - Similarly, 9u n _is a quotient of 9u n22 , 
where fj(u) = 0(74 n _i, n ). 


Proof of Lemma \2.1\ The proof closely resembles Proposition 4 of Bump, Friedberg and 


Ginzburg [BFG06] . We prove only the first assertion, the second is symmetric. For 1 < j < 
u-l, let Vj = Uj trL -jnMj_i tn _ j+i (e-g., V = Ui >n -i and n"=i Vj = N n ). Let 0 be the character of 

nil Vi defined by 0(u) = 0(£ 3 i=1 v iM i). Then (9 Vl ^)v 2 ^ = 9 Vl v 2 ,il> and GL n _j_i < M ]+ \^ n -j-\ 
stabilizes 0 and acts transitively on the nontrivial characters of Vj+ 1 . Hence it suffices to 
show 


9v x v 2 ,ip - 0 , 

as this would imply that the action of U 2, n -2 on 9u ln _ lt i/j is trivial. Suppose otherwise and 
let j > 2 be maximal such that 

( 2 . 10 ) 

Since for n>3, 6 does not have a Whittaker model, we must have j <n - 2. Then 

= ^ni =1 Vi,ip)yj+i = = 8Q j+ln _ j _ 1 (9j+i®9 n -j-i)N j+1 ,i>, 

where the last equality follows from (12.81) . As in the proof of Theorem 12.61 above, we use 
Claim 12.31 and restrict 9 j+ i®9 n - j_! to GL J+1 x GL ■_!, and since (9j +3 )j = 0, we obtain 
(9 j+1 ®9 n - j - 1 ) Nj+1 ^ = 0 contradicting (12.1 OF □ 

3. Twisted Jacquet modules of exceptional representations 


In this section we study twisted Jacquet modules of 9 - 9 ri)Xn . Let U = U n _ k ,k, f° r 0 < k < n. 
The Levi subgroup M = M n _ kk acts on the set of characters of U , with min(?7 - k,k) + 1 
orbits. We choose representatives 


7 -J 00 — 0 ( r U"n-k—i+l,n—k+j—i+l '), 


i= 1 


0 < j < min(?7 - k, k). 


For example when n- 5 and k = 3, 0i (u) = 0 ( 742 , 3 ) and 02( 74 ) = 0(742,4 + T4i j3 ). When n 
0 fc ( Ik h ) “ 0(tr(u)). The stabilizer of 0j in M is 


Stn,fc(0j) _ " 


lb V 
C 


c y 
d t 


b e GL n — k -j, c g GLj, d g GL k ~j * 


2k, 


Let GLj denote the embedding of GLj in St n ,fc(0j) via the coordinates of c. The restriction 
of the cover to GLj gives a trivial double cover. In fact by (12. ip if c,c! £ GLj, 

er(diag(/ n _fe_j, c, c, 4_j), diag(/ ra _ fe _j, c', d, h-j)) = (det c, det c') 2 . 
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Hence any genuine representation of GIL,- takes the form tt ® 7 ^, where 

7 T ® 7^(es(diag(/ n _ fc _j, c, c, 4 -j))) = e7^(det c)tt(c), e e /i 2 . 

The following theorem was proved by Gelbart and Piatetski-Shapiro [ GPS80 ] (Theo¬ 
rem 2.2) for the case k = 1. We extend it to any k. 


Theorem 3.1. Assume n = 2k. Then 9u,ip k is one-dimensional and in particular, irreducible. 
The action of GL fc on this space is given by the character 

es(diag(c,c)) ex 2 (c) 7 V , j( _ 1 ) fc-i(detc), cc GL fc . 

Remark 3.2. It follows that the space of functionals X on 9 such that X (9(u)ip) = ifk(u) X(<p), 
where <p belongs to the space of 9 and u e U, is one-dimensional. In the non-metaplectic 
setting, if tt e Alg GL n and tt u,^ k is the trivial character, t r admits a Shalika functional (see 
Section WTA) . Since x 2 corresponds to x 2 ®Tit ( as explained above, because the cover is trivial), 
it is natural to call X a metaplectic twisted Shalika functional, or simply a metaplectic Shalika 
functional when x = 1- Of course it implies an embedding 9 c Inaff'ff ( x 2 7 i/>, (- 1)- 1 ® 4fc)> 
which is a metaplectic twisted Shalika model. We will use this observation in Section ^ One 
can expect uniqueness of this model, as in the non-metaplectic setting. 


Proof of Theorem \3. 1[ We use induction on k. For k = 1 the result is known: one-dimensionality 
holds because then 9 is generic, and the action of GLi is es(diag(c, c)) ex 2 ( 0 ) 7 ,/, (c) by 
[GPS80 ] (Theorem 2 . 2 , see also |Kab01] Lemma 5.3). 

Denote the elements of U by 


v(v 1 ,v 2 ,v 3 ,v 4 ) 


f 1 V\ v 2 ' 

4-1 v 3 v 4 

1 

^ 4-i , 


By definition ipk(v(vi,v 2 ,v 3 ,u 4 )) i4h)- Let 


H 3 = {n(0,0,u 3 ,0)} < V, V 1j2 ,4 = {v(vi,v 2 ,0,v 4 )} < V, 



According to the local analog of “exchanging roots”, proved by Ginzburg, Rallis and Soudry 
[ GRS99] (Lemma 2 . 2 , stated for unipotent subgroups of symplectic groups, but the argu¬ 
ments are general and hold in our setting), in AlgR 17 . 4 , 


(3.1) 


9ux k = 9v 1>2A xE^ k - 


Indeed, it is simple to check that the list of properties stated in the lemma holds in this 
setting (in the notation of [ GRS99] . C = Vi, 2 , 4 , X = V 3 and Y = E). 

Let 


1 

£ 

4-i 


W = 


\ 


4-i ) 


£= ( — If" 1 . 
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Then < W \0 V2 ^ E ^ k ) = Ou'tf, where 


r 

f 1 «i e v 2 s 

> 


1 0 0 



Ik -1 V A 


< 

\ h-1 ) 

> 




We chose w e SL n , in order to easily appeal to the formulas of Banks, Levi and Sepanski 
[ BLS99 J for computing conjugations of torus elements by s(w) (see below). The actual 
choice of section s does not matter for the conjugation, henceforth we simply write w. Since 
U' = U x . /n -x x U k -i,k-i, where U k -i,k-i < M 2 , 2 (k-i), and = i>(^), Lemma | 2 T] implies 

9u’,i>' = where = ip(£u lt2 ) and in particular, ip°\u 2in _ 2 = 1. This 

Jacquet module factors through the Jacquet module with respect to U 2)n - 2 and the trivial 
character, hence by ( 12 . 8 ft it is equal to 

^Q 2 ,n- 2 (^ 2 >X>7l®X7^-2,X>72)A r 2x[/ fc _i,A ! _i,V'(e-)®hi ! -l' 

Here 71 and y 2 are arbitrary. We restrict to GL° x GL°_ 2 . Since (9 2 ®9 n - 2 ) a - 0° ® 0°_ 2 
(Claim [23]), we obtain 

(3-2) 2,71.-2 (() n 2 ,y(£-))° ® ((^- 2 ,x, 72 ) 14 - 1 ,fc-i,hfc-i) D - 

By the induction hypothesis both spaces are one-dimensional, hence 9u,yi k is one-dimensional. 

Regarding the action of GL/., it is of the form rj <£> 7 ^, for some character 7] of F* . hence it 
is determined by its restriction to a maximal torus. 

We describe the isomorphism (13.11) . in order to understand how this action is transferred 
from du,ip k to (Vi, 2 , 4 XiE,v>k- The isomorphism was given in |GRS99] (proof of Lemma 2.2) using 
the following chain of isomorphisms: 

(Ovi ,2,4 ,ipk ) E -(ind^(^J) B 

Ou,tk- 


The first step was to define a mapping 0y 124) ^ fc -> ind u* E (0u,ip k )- For <p in the space of 9, 
f(<p + 0(Vi, 2 , 4 ,i/>k)) bi the space of md%* E (0u^ k ) was given by 


= 9(x)ip + 9(U,i/j k ), xeUxE. 


This mapping was extended to a mapping 6y 124X E,ip k -*■ (ind *u* E (9u,ip k )) e by 
f(<P + e(y 1A iE,ih))(x) = f(<p)(x) + (ind v» e (9u^))(E). 

To obtain an element in 6C,y fc , integrate f(<p + 9(V\ y2 ^E ^ k ))(x) over E. Altogether 

<p + 9(V li2A E,i/j k ) h » f 9(e)if de + 9(U^ k ). 

J E 

Let Tfc denote the image of T k in GIL*,. Since T k normalizes Rig, 4 , E and U, and stabilizes 
i/) k , the isomorphism (13.11) extends to AlgT*,. 

Let t = diag(H, t 2 , t\, t 2 ) e T n with U e F* and t 2 e T k _i (t is a general element of T k ). Since 

( 3 . 3 ) 9(s(t))(f + 9(V lt2A E,7p k ) ^ ^ n 2 ( w t)6(s(t)) f 0(e) ip de + 9(U,ip k ), 

J E 

the action of T k on 9u,ip k is transformed by (13.11) to w ' 2 multiplied by the action of 
T k on 9v 2 ,3,4^E^ k - The latter is given by 9 V23A ^ E ^ k (5(t)) = 9u'^'( w s(t)). Using [BLS99] 
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(Section 2 Lemma 2, Section 3 Lemmas 3 and 1) we see that 

w s(t ) = cr(w,t)s( w t ) = (ti,deU 2 ) 2 (dett 2 ,detf 2 ) 2 s(diag(ti,UT2,£2)) 

= (ti,dett 2 ) 2 (dett 2 ,dett 2 ) 2 s(diag(ti,ti,/ 2ri _ 2 ))s(diag(/ 2 ,t 2 ,t 2 )) e GL° x GL°_ 2 . 
According to the induction hypothesis 

(On -2 (s(diag(t 2 , t 2 ))) = y(det t 2 2 ) 7 ^ _ £ (det t 2 ) 

= x(det f 2 2 )(det t 2 , det t 2 ) 2 “ 2 7 y(det t 2 ), 

therefore 

9u,i’ k (s(t)) = 6u\^’( w s(t)) = (y 2 7 v , )( _ 1)fe _i)(detdiag(t 1 ,t 2 )). 

Here we used (j2.2j) and the fact that (-l,a) 2 = (a,a) 2 . Note that the modulus character 
appearing in (13.2p was cancelled by the twist of the action due to (13.31) . □ 

Using this result we can compute all the twisted Jacquet modules for maximal parabolic 
subgroups. 


Theorem 3.3. Let 0 < j < min(n - k, k). We have 

Here the metaplectic tensor is a representation of p~ 1 (GL n _ k _j x GL k _ 3 ), where GL n -k-j x 
GLfc_j is embedded in GL n through St n ,fc('0j)j A/ie pseudo-characters 7 j ; 1 < z < 3, are arbi- 
trary; the pseudo-character 74 ) is arbitrary when n is even, and uniquely determined by 

(3.4) l{3)( z ) = l( z )hi>A-iy- 1 ( zj )i VzeF*, 

when n is odd; the right-hand side is regarded as a representation in AlgSt by extend¬ 
ing it trivially on U n ^k-j, 2 j,k-j- 

Remark 3.4. Note that 7 ^) is really an (n-2j)-pseudo-character: when j is even, 7 y i (-ip-i (U) 
1 and [(n - 2j)/2\ = [n/2\ (2), while if j is odd, [(n-2j)/2\ = [(n-2)/2\ (2). 

Proof of Theorem \3.3[ For j = 0 this is (12.81) . assume j > 0. We can also assume that if 
n = 2k, j < k, otherwise there is nothing to prove. The assertion holds for n - 1,2, assume 
n > 3. First we claim that U n -k-j, 2 j,k-j acts trivially on 0u^i> : , ■ 

Claim 3.5. 6^ = 9 Un , . . . k 

The proof is given below. Assuming this, according to (12.81) and (12.71) . 

= ^Qn-k-j,2j,k-j (^-fc-J.X.71 ®X7 (^2j,x,73 ®X7 '72 ) ) ■ 

Here U 2 j,k-j < GL k+j < M n _ k _ jMj . Hence, noting Vhlc 7 .- fc -., , 2 ., = 1, 
where U jd < GL 2j < M n _ fc _ J; 2j - fc _ r 

To compute the Jacquet functor with respect to U ]tJ and fjj, we use the Geometric Lemma 
of Bernstein and Zelevinsky 1BZ7 7 (Theorem 5.2). According to the associativity of the 
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metaplectic tensor f |Kab01j Proposition 3.5), and the fact that induction of a semisimple 
representation from a subgroup of finite index is semisimple, 


@n-k-j,x, 71 ®X7 (^2j,x,73 ®X 7 '^ fc ~. 7 >X >72 ) 

= i 1M V 1 (GL n ^_/xGL° xGL fc _j) ^®X7"^-7,X,72) ® °0> 


where a e AlgGL 2j is a suitable irreducible summand of ^ X73 , which depends on 7 " pre¬ 
cisely when n is odd. The double coset space GL 2 j \GL 2j /GLj contains [F* : F* 2 ] rep¬ 
resentatives and we may take them in the form g z = diag (zlj,lj), where z varies over a 
set of representatives of F* 2 \F*. These representatives normalize GL n -k-j x GL 2j - x GLk-j, 

GL n _ fe _j x GLj x GLand Ujj , and he., 9z i ) j ((G)) = , 0(^'tr(n)). Therefore 


($ri-A 


-JiX>7l 


< ^ > X'y(^2j. 


X>73 


X7 , ^ - iiX>72 


)) 


Uj,j ■t'tpj 


is glued from representations 


• jP (GL n _fc_j xGLj xGLfc_j ) 

V 1 (GL n _*._j- xGL° xGL fc _j-) 


(^ z ( 71 ^x 7 "^- 3 ^ 12 ) ® ZtF \F . 


Since GL° = GLj, GLj and p 1 (GL n _fc_j x GL^.j) commute whence each of these represen¬ 
tations equals 

9z ($n-fc-yx,7l®X7"^ fc -j,X,72) ® 9 ~ a Ujj ,ip(z-)j ■ 

The block-compatibility formula (12.11) implies (see [KabOl] p. 748) 

9z ® Xl" ^ k- i :X:1‘2 ) ~ Xz ® ((^n-fe-J,X)71 ® X7”^^-J;X;72 ) 1 


where is the non-genuine character of M n ^-j,k-j given by \y(c/) = (det g, 27 ) 2 . This is just 
a twist of the metaplectic tensor and will eventually be reflected on the pseudo-character 
changing from 7 " to 77 -). We show that 9 * <J Ujj ^z-) j vanishes for all but one representative 
Zq, for which it equals ((G/,*, 73)17 , 7 , (both representations belong in AlgGLj). This will 
complete the proof. 

In general let 7 r € Alg irr GL 2 j be genuine. Then 7 t d is the direct sum of [F* : F* 2 ] irreducible 
representations 77 and moreover, for each fixed summand p - 77 , 7 r = ind^n (p) ( [KabOl] 

GL 2 j 

Proposition 3.2). Now an application of [ BZ77J (Theorem 5.2) similar to above implies 
that 7 TUjjdj is glued from representations 9 zi pu :i:i . t j)(z-)p where z varies over the different 

square classes of F*. Furthermore, since if) and ^(y 2 -) are conjugates in GL 2j - (use g y - 2 ), 
the spaces PUjj,^ and Pu jj ,p{y 2 -) j are isomorphic. Assuming iTu :j ,is one-dimensional, these 

observations imply that for each p, ffz ° pu ;j jA i)(z 0 -)j * 0 for some Zo and furthermore, Zq is unique 
modulo F * 2 with this property. 

By Theorem 13.11 ( 027 ,*, 73 )^' j,pj is one-dimensional, hence for each summand a there is a 
unique representative z 0 e F* 2 \F* such that 9 z °cr UjJ ^( Z0 .) j = (0 2 j, x , 73 )u j>jt ii> r 
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Proof of Claim [X5l Let V = U n -k-j,j be embedded in St n,fc(V , j) as the subgroup of matrices 
corresponding to the coordinates of v and denote its elements by 


v(v 1 ,v 2 ,v 3 ,v 4 ) 


ln-k-j-1 P.3 

1 Vi 

1 


P2 ' 

V 4 


J i-1 

The group 1/ is abelian and normalizes U (recall that U = U n _k,k)- First we show 


(3.5) 


Ou,iij - &uv,ipj ■ 


It is enough to prove {Ou 4 p :j )v 4 i = 0 for any nontrivial character y of V. The group 
St n ,k('f’j) n Mn-k-j,j,j (he., the coordinates of b and c in St n ,k('f , j)) acts on the characters 
of V and we may assume that \x does not depend on the coordinates of v 3 and v 4 , and 
K v ( v u 0 , 0 , 0 )) = f’(vi). 

As in the proof of Theorem 13.11 and with a similar notation, if 


V 3 = {u(0,0,u 3 ,0)} < V, 


hi, 2,4 = {V(V!,V 2 , 0 , V 4 )j < Id, E = Un-k-j- 1,1 < GL n-k-j, 


where U n _k-j- 1,1 is the unipotent radical opposite to U n -k-j-i,i, Lemma 2.2 of Ginzburg, 
Rallis and Soudry [GRS99] implies the following isomorphism in AlgVi, 2 , 4 , 

Conjugating the right-hand side by ( 7 Ij+1 ), we see that ( 6 , c/ 1 a )v 124 eu is a quotient 
of 


V'(w) = 1p(ui, 2 ), 1p'(u) = V’(«2,n-fc+l), 

where if was obtained from y\{ v ( Vl , 0 , 0 , 0 )} and ip' from tpj. This space vanishes according to 
Lemma [2771 Hence (0[/,y- 7 )v> = 0 and (13.5p is proved. 

Now let Y = Uj t k-j be embedded in St n ,k{ipj) as the subgroup of matrices corresponding 
to the coordinates of y, 


V = 2/(2/i, 2/2,2/3, 2 / 4 ) 


/ I 


3 - 1 


2/4 

2/i 

1 


2/2 
2/3 

h-j-i ) 


We show {0uv,ii>-) y,h = 0 for any nontrivial character y of Y. The proof is a repetition of 
the argument above, this time /1 can be assumed to depend only on y\ and y 2l we take 
E = Ui’k-j-i and conjugate using ( Ij+1 ) (embedded in the bottom right corner of GL n ). 

Then the result follows from Lemma I27T1 (flrr 1 1 j, is a quotient of 0u n _ 22 ). 

Summing up, we have shown ( 9u ,^-) = ( Ouvy, ipj) and since UVY = U n -k-j,j,j,k-j , the claim 
is proved. □ 


□ 
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4. Distinguished representations 


Let r e AlgGL n be admissible and assume that r admits a central character uj t . Given a 
pair of exceptional representations 9 and 9' of GL n , we say that r is (9, ^-distinguished if 

(4.1) Hom G L n (^®^,7 v )tO. 

In light of Claim 12.51 the spaces 9 X: 7 as x and 7 vary are twists of each other by characters 
of GL n . Since 9 xn = x0g 7 , it is natural to £x x = x! = 1- Dehne r to be distinguished, if it is 
(0g 7 ,#g 7 > ^distinguished for some pair of pseudo-characters 7 and 7 '. 

A distinguished representation r always satisfies oof = 1. In particular if r e Alg esqr GL n 
is distinguished, it is already unitary whence r e Alg sqr GL n . If n = 1, these are the square- 
trivial characters of F*. 

The following simple claim explains the motivation for removing any specific choice of 
pseudo-characters from the definition. Note that when n is even, the pseudo-characters are 
redundant because they do not affect the exceptional representations (see Section 12.51) . 


Claim 4.1. If n is odd, for any square-trivial character rj of F *, r is distinguished if and 
only if r/T is. 


Proof of Claim [J~l\ For a fixed pseudo-character y 0 , the set 7/70 with 7 varying over the set 
of pseudo-characters, exhausts all square-trivial characters of F*. Then if r is (#g 7 , 0g 7o )- 
distinguished and 7 = Yho, 7 ^ 1,70 = ^ 1 , 7 ' ( see Claim 12.51 r](z) n = rj(z ) when n is odd) and r/r 
is (#1,7, #i i7 '^distinguished. □ 


This claim is also useful for the following observation. Assume we have, for an admissible 
representation 7y®r 2 e Alg M n - kik (he., 77 e AlgGL n _fc, 77 e AlgGL^ and both are admissible), 

ftom Mn _ k k ((9 n _ k ®9 k ) ® {9' n _ k ®9' k ),Tf ® r 2 v ) * 0. 

According to Corollary 12.41 if k is even, 77 is distinguished. I 11 the odd case, the corollary 
only implies that ?/r 2 is distinguished for some rj with rf - 1. Then by Claim 14.11 77 is 
distinguished also when n is odd. Similarly, we deduce that 77 is distinguished. 

The following lemma is the application of the results of Section [3] to the study of (14.11) 
and will be used repeatedly below. 


Lemma 4.2. Let 9 = 9 n ^ n , 9' = 9 H} gy and 7y®r 2 e Alg M n _ kk be an admissible representation. 
Assume 


( 4 - 2 ) Hom M n _ fc , fc ((0 ® 9') Un _ kk , ® rf) ± 0. 

Then for some 0 < j < min(n - k, k), 

® 1, 0 5 q!1-M 2)C-.^) V ) * 0 - 

Here L nk j = GL n - k -j x GLj x GL k ~j, where GLj is the diagonal embedding of GL j in 
St n -k,k (Vh")/ £n,k,j is the following representation of p~ 1 (GL n ^k~j x GL k _j), 


^n,k,j ( 9 n-k-j, 1,71 ®7(,-) 9 k-j, 1,72 ) ® (fln-fc-J,! 


V 

,7 1 7 U) 


9k-j,i,y 2 ) 


where 7 g 7 ' are arbitrary, 7 ^) is given &g (13.41) with respect to 7 and the character if, and 
7 C is given by (13.41) with 7 ' and ; 1 is i/ie trivial character of . 
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Proof of Lemma \f-2\ By an analog of the Geometric Lemma of Bernstein and Zelevinsky 
( [BZ77] Theorem 5.2 and [BZ76j 5.9-5.12), in A\gQ n _ ktk , 6 is glued from 

ind-" - *’* . TT ( 6u uutbj), 0 < j < min(n - k, k). 

Regarding the notation, see Section [3] According to Lemma 12.21 ( 6 ® &')u n _ k k is glued in 
Alg Q n -k,k from 

(43) ® e u„_ Kt ,rp' min ( n - k ’ k y 

Note that U n -k,k acts trivially on (14.3[) and we regard it as a representation in Alg M n - k ,k- 
Also recall that by Theorem 13.31 

®Un-k,krtf = 0 (0 2 Lh'y3)u j , j ,iPj 1 - 

By Theorem 13. H and because y/.a.;.- 1 = 1, ( 6 , 2 j,i, 7 3 )c/ ) - 3 -,^®(^ 2 j,i, 7 ' )u ■,V ' t1 is the trivial character 
of GL r Set V = Un—k—jjjk—j. Then 0 Un _ k k ^ ® 0' Un fcfc ^_i e Alg(L njfcj - x V) and is trivial on 
V. For g e L nkj , let mody(g) be defined by 


J ~ f( 9 'u) dv = mody(^) f(v) dv. 


This is the modulus character from [ BZ77] (p. 444). Then mody-(g) = 5Q n k ] . 2 . fc (g). There¬ 
fore f!4.3p becomes 


ind' 


Q n—k ,k 


L nk jv( m0( ^V (£ra,fcj®l))- 


Assumption (14. 2 p implies that for some 0 < j < min(n - k, k ), 

Hom Mn _ fc fe (ind^- fc ’ fc y ( m ° d J/ 2 (^ )fcJ ® 1)),*^^ ® r 2 ) * 0. 

The left-hand side equals 

Hom M„_ fc , fc (R ® t 2 , I nd?^V( m odv 2 ^ 2 fcifc (^n ) fcj ® l) v )) 

= H ° m W M ( mod V /2 ^l feifc ( T 1 ® A)v, (fn.kj ® 1) V ) 

- (£n,k,j ® 1, (^H 2 fc _ W ( r l)^n- fc -jJ 0 5 Q^( T 2 )g,^) V )- 


□ 


Our hrst objective is to describe how distinguished representations are constructed. In 
|Kap l4b| we proved the following heredity result: 

Theorem 4.3. 7 |Kapl4b| Theorem 3) Let ti®t 2 e Alg M n _ k ,k- U T i an d T 2 are distinguished, 
T\ x r 2 is distinguished. 


This does not exhaust all distinguished induced representations, as can already be observed 
in the case of n - 2 and principal series representations: if r is any character of F*, r x r _1 
is distinguished. The proof is contained in |Kapl4b|| (Claim 4.4), but was only reproduced 
from the arguments of Savin [Sav92j . We extend this result to the following theorem. 

Theorem 4.4. Let r e Alg irr GL fc . Then r x r v (= Ind Q^ 2k (^nl k r ® rV )) a distinguished 
representation of GL 2k - 
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The theorem essentially follows by exhibiting a functional in 

Hom M fc , fc ((0 ® °') Uk ,k^Qlj W ® r )- 

We will use the metaplectic Shalika model of 9 obtained in Theorem 13.11 to write an ele¬ 
ment of ( 9 ® Q')u k k as a locally constant function on GL 2 /c, bounded by a Schwartz-Bruhat 
function. The integral of this function against a matrix coefficient of r will be absolutely 
convergent in some right half-plane, since it can be bounded using the well-known zeta in¬ 
tegral of Godement and Jacquet [GJ72j . Then one can obtain meromorphic continuation 
using Bernstein’s continuation principal (in |Ban98j ). 

Proof of Theorem \4-4\ Put Q = Qk,k, M = M*,^, U = Uk,k and G = GLfc. Denote by Cff(G) 
the space of complex-valued locally constant functions f on G that are bounded by a positive 
Schwartz-Bruhat function in S(F kxk ), that is, there exists (f e S(F kxk ) such that \f(g)\ < <f(g) 
for all g. The group M acts on this space by diag (a,b)f(x) = /(fc _1 xa). Let 0 = 0 2k , i , 7 and 
9' = # 2 fc,io" (^ ie pair 7 and 7 ' does not matter). By Frobenius reciprocity, we need to prove 

(4.4) Hom M ((# ® 9')u, 5q 2 t v ® r) + 0. 

Denote I (if) = Ind~^ 2fc ®if k ) and i(if) = ind?^ ( 7 ^ ®ifk), where G is the 

diagonal embedding of G in M. According to Theorem 13.11 (see Remark 13.2p . 9 c I (if). 
Additionally 9(U) c 9 in AlgQ and since ind~ (9uib k ) c 9(U), the theorem also implies 

i(if) c 9 (in AlgQ). When we apply Theorem 13.11 to 9' using if _1 , we obtain i(if _1 ) c 6' c 
I(if~ l ). First we claim: 

Claim 4.5. There exists L e Hom^((0 ® 9')u, Cf°(G)), which restricts to an isomorphism 
(i(if) z(' 0 _1 )) C 7 =S(G) in Alg M. 


Now let p ® ip' belong to the space of 9 ® 9'. Also let h be a matrix coefficient of r. For 
seC, consider the integral 


(4.5) 



L(ip®ip')(g)h(g)u‘ m (g) dg. 


Since L(p ® p') e Cf°(G), there is some s 0 e ® depending only on r, such that (14.51) is 
absolutely convergent for all s with 93(s) > So- This follows immediately from the convergence 
properties of the zeta integral J G <f(g)h(g)v s (g)dg of Godement and Jacquet f |GJ72j p. 30). 
In this right half-plane (14.5ft defines an element in 


(4.6) 


HoniAf((0 ® 9')u, Sq 2 u s t v ® u s t). 


Also since L : (i(if) ® 5(G) is an isomorphism, we can choose data (p,p',h) 

such that (14.51) is absolutely convergent and equals 1, for all s. I 11 order to deduce meromor¬ 
phic continuation, we also need to show that (14.61) is at most one-dimensional, except for a 
finite set of values of q~ s , where q is the order of the residue field of F. In fact, we prove the 
following more general statement. 


Claim 4.6. Let T\ <8> r 2 e Alg irr M n -k,k■ Outside of a finite set of values of q 

Hom M n _ fc , fc ((0 ® 0') Un _ Kk , 5^l_ k k 10~ S T\ ® V s t 2 ) 


is at most one-dimensional. 
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Now according to Bernstein’s principle of meromorphic continuation and rationality (in 
[Ban98] ). the integral (14. 5 p has a meromorphic continuation to a function A(<£>® < p',h,s ) in 
C (q~ s ), satisfying the same equivariance properties with respect to M, and not identically 
zero at s = 0. Assume A(-,-,s) has a pole at s - 0, of order r > 0, then lim s _ 0 s r A(-, •, s) is 
hnite and nonzero. Now (I4.4j) follows and the theorem is proved. 


Proof of Claim \4-5\ Regard G{- GL fc ) as a subgroup of GL 2fc via the embedding i{x) = 
diagIn general, any / e I {if) defines a locally constant function on G by restriction. 
According to Lemma 3.1 of Friedberg and Jacquet |FJ93j . which can easily be verihed in 
our setting, this function is bounded by a positive Schwartz-Bruhat function in S(F kxk ) (the 
more general statement in |FJ93] also holds, but will not be needed here). 

Define a mapping L : I {if) ® /(t/’ - 1 ) C£°(G) by 

L{f®f){x) = f{£{x))f{e{x)). 


Since = 1, we have /(diag(x, x)g)f'{di&g{x, x)g) = f{g)f'{g)- Hence L intertwines 

the action of M and belongs to 

Horn M (/(^)®/(r 1 ),C'r(G)). 

Observe that L factors through {I {if) ® I{if~ l ))u- Indeed if Y, a fa ® fa lies i n H ie space of 
(I{if) ® /(V ;_1 ))(G), Lemma [2.11 implies in particular that for some compact U <U, 

(E/“® = f du = 0, VreG. 

a JU a 

Here we used the fact that for x 0 e G such that p(x 0 ) = diag (x,Ik) = £{x) ( p - the projection 
GL 2 fc -> GL 2 fc, see Section [24]) , f a {x o ( /fc 4 )) = f’k{xu) f a {x 0 ), and assumed f u du = 1. 

Moreover, Lemma 12.21 and its proof show that {i{if) ® i{if~ x ))u = ind^ t/ (l) = S{G) and 
the restriction of L to i{if) ® i{if _1 ) c I {if) ® I{if~ l ) defines an isomorphism in 

B.om M {{i{if) ® i { l f~ 1 ))u,<S{G)). 

The claim follows because 9 ® 9' c I {if) ® I {if" 1 ). □ 


Proof of Claim f .6 By virtue of Lemma 14.21 and using the same notation, it is enough to 
prove that for all 0 < j < rnin(ri - k,k), except for a hnite set of q~ s , 


® L ( vS ( r i V )F 


J n-k-j,j 


y 1/2 


wk ( n = 0 , 


and for j - min(n - k , k) this space is at most one-dimensional. 

We show that this holds for each pair of irreducible subquotients Qi®Q 2 of . ( r i)u n _ k _ :j , 

_ 1 /o 

and Q 3 ® ^4 of k _.{rf )u jk . r For these subquotients, this space vanishes unless both of the 
following spaces are nonzero: 


(4-7) Hom G L n _n_jxGLj_j {f,n,k,jt v *Qi ® qI), 

(4.8) Hom G Lj {v s @ 2 , v s Q 3 ). 

If j < k or j < n-k, Corollary 12.41 (restricting (14. 7 p to GL°_ fc _j x GL k _j and GL n _ fc _j x GL°_j) 
implies that (14. 7 p vanishes unless V s Q\ and v~ s Q± are distinguished, which is false for almost 
all values of q~ s (simply considering the central characters). 

The remaining case is when j - k - n - k, then (14. 8 p becomes HoniGL i .(^ s 'r 1 v , z/ s r 2 ), which 
is at most one-dimensional. □ 
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□ 


Theorems 14.31 and 14.41 imply the following corollary. 

Corollary 4.7. Let a = (aq,... ,a m ) be a partition of n and T\ <£>... <S> r m € Alg M a . Assume 
that if oti is odd, Ti is distinguished, and if a.i is even, r % is either distinguished, or equals 
Ti t o x for some r,; ; o e Alg irr GLq,.^. Then T\ x ... x r m is distinguished. 

Here and onward we assume that A is a field of characteristic 0. This is because we rely 
on several results not proven for fields of nonzero characteristic, e.g., certain functoriality 
results. 

The following remark will be used in our analysis of square-integrable distinguished rep¬ 
resentations. 


Remark 4.8. Let r = {\vA~ l )l 2 p^d- 1 )! 2 p]) 1 , where p e Alg cusp GL fc is unitary and l > 1 
is an integer (see Section [Ql) . Put n - Ik. The computation of L(s,r, Sym 2 ) by Shahidi 
|Sha92j (Proposition 8.1) implies the following characterization for the existence of a pole of 
L(s,r, Sym 2 ) at s - 0: 

(1) If n is odd and p is self-dual, this is always the case. 

(2) When n is even and l is odd, L(0,t, Sym 2 ) = oo if and only if L(0, p, Sym 2 ) = oo. 

(3) When n and l are even, L(0,r, Sym 2 ) = oo if and only if L(0,p, A 2 ) = oo. 

In particular, for a pole to exist p (equivalently, r) must be self-dual. 


The next globalization lemma will be applied, to deduce that square-integrable repre¬ 
sentations r such that L(s,r, Sym 2 ) has a pole at s - 0, are distinguished. It may also 
be of independent interest. As mentioned in the introduction, there is an assumption here 
concerning the quasi-split case, see Remark 14. 1 II below. 

Lemma 4.9. Let tt € Alg sqr GL n . Assume that L(s,7t,R) has a pole at s - 0, for R = Sym 2 
or A 2 - Then there exist a number field with a ring of adeles A and a global cuspidal repre¬ 
sentation n of GL n (A), such that for a sufficiently large finite set of places S, L s (s, n,R) 
has a pole at s - 1, and at some place v, = 7 r. 


Proof of Lemma \4-9 


Consider the local functorial lift of |CKPSS04j . Let 


G n = 


S02m+1 

Sp m 

S02 m 


R = A 2 , n = 2m, 

R = Sym 2 , n = 2m + 1, 
R = Sym 2 , n = 2m. 


Here the even orthogonal group is quasi-split (split, or non-split but split over a quadratic 
extension). We claim that there is a generic square-integrable representation ti' of G n , such 


that 

(4.9) L(s,n' X g) - L(s,n X g), e(s, n' X g, if) = e(s, n X g, fj), 


for any generic g e Alg irr GLfc and k > 0. Here the L and e-factors are defined by the 
Langlands-Shahidi method [ Sha90] . 

Indeed, if R = A 2 this follows immediately from Theorem 2.1 of Jiang and Soudry |JS04] . 
Assume R = Sym 2 . Then if n is supercuspidal, this follows from |ACS14] and |PR12] (the 
appendix). In fact, the local functorial lift of an irreducible supercuspidal generic represen¬ 
tation, of a quasi-split classical group, to some GLat was detailed in [ ACS 14] (Theorem 3.2). 
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Given that, the results of Jiang and Soudry in the appendix of [PR12] show that the lift 
from quasi-split S0 2m is surjective onto the supercuspidal representations of GL 2m . 

Regarding the remaining cases where R = Sym 2 : the existence of n' when n is odd, is 
contained in Theorem 4.8 of Liu [Liull] : if n is even and the central character c o n = 1, this 
was proved in Theorem 4.8 of Jantzen and Liu [ JL14j (in [JL14 ] only the split case was 
analyzed, when n = 2m and c + 1, S0 2m will be non-split but quasi-split). We prove the 
last case in the following claim. 


Claim 4.10. A representation it' with the above properties exists also when n = 2m, R = 
Sym 2 , 7r is not supercuspidal and oj n ± 1. 


Additionally we take p e Alg cusp GL n such that L(s,p, R) has a pole at s - 0. By |JS04j . 
[ACS 14 ] and [PR12 J (the appendix), there is a generic supercuspidal representation p' of G n 
satisfying equalities similar to (14.91) . 

Now according to Ichino, Lapid and Mao [ ILM14 J (Corollary A.6, which actually holds for 
the types of G n stated above, when G n is split, by replacing Proposition A.5 with |CKPSS04] 
Corollary 10.1), there exist a number field with a ring of adeles A and a cuspidal globally 
generic representation II' of G n ( A), such that IL^ = tt' and H' V2 = p' for some pair of places 
V\ and v 2 . 

Next we apply the global lift of [ CKPSS04 , ICPSSllj and obtain an automorphic rep¬ 
resentation II of GL„(A) such that Zv* 5 (s,IT,R) has a pole at s - 1, assuming S is large 
enough. Moreover, the global lift preserves the y-factors at all finite places v ( |CKPSS04] 
Proposition 7.2), hence 

y(s,n^ X p,-0) = 7 (s,n;. X p,-0), Vpe Alg cusp GL fc , 1 < k <n - 1, i = 1,2. 
Together with the fact that the local lift preserves L and e-factors, we see that 
7 (s,n ui x e ,$) = y(s, 7r x y(s,n. 02 x = y(s,px 

for all q £ Alg cusp GLfc and 1 < k < n - 1. By virtue of the local converse theorem for GL„ 
( jHen93l after Theorem 1.1), this implies Ll 1;i = 7r and LR 2 = p. Because p is supercuspidal, 
we deduce that LI is cuspidal. 


Proof of Clam \%GTty We adapt the proofs from [ JS04 ] (Theorem 2.1) and [JL14] (Theo¬ 
rem 4.8). We are interested in a simpler result, concerning only square-integrable represen¬ 
tations. Our assumptions on 7r are that n = 2m, L(s, 7r, Sym 2 ) has a pole at s - 0, 7r is 
not supercuspidal and uy- + 1. Hence 7r = (\y~ l E, v l e\Y, where e e Alg cusp GL fc is self-dual, 
0 < l e Z, k is even, 2m = (2 1 + 1 )k and L(s,e,Sym 2 ) has a pole at s - 0 (see Remark 14.81) . 

According to [ACS14j and [PR12j (the appendix), there is a generic supercuspidal repre¬ 
sentation e' of a quasi-split SOk satisfying 

(4.10) L(s,s' X q) = L(s,ex g), e(s,e'X g,t/}) = e(s,ex 


for any generic g € Alg irr GL fc and k > 0. Then {{ye, is a representation of GL i k xSOk. 

Consider the representation ([ue, ^e])‘xe' of S0 2r m parabolically induced from ([ue, u 1 e]Y ® 
e'. Let 7 r' be the unique irreducible generic subquotient of (\ye, v 1 e]Y x s’ . One can show 
that 7r' is square-integrable (here the root system of S'0 2m is of type i; the arguments 
of |Mui98] Section 2 and [Tad98bj . for S0 2n + i, can be slightly modified to apply to S0 2m ; 
in the notation of |Mui98j the pair (e,e') satisfies (Cl)). Now the proof of the preservation 
of local factors, i.e., (14.91) . follows exactly as in [ JS04| (proof of Theorem 2.1), using the 
multiplicativity of the y-factors and (14.101) . □ 
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Remark 4.11. In one case, namely when n = 2m, R = Sym 2 and co n ± 1, the classical 
group S02m is non-split (split over a quadratic extension). In this case, the lemma is valid 
under two mild caveats. First, the functorial lift in the quasi-split case was only described 
globally flCPSSllj ). leaving out the local lift, which we use. Second, in appealing to |ILM14j 
(Corollary A.6), we need the following result obtained in [CKPSS04] (Corollary 10.1) for split 
groups: for a globally generic cuspidal representation II' of G n (A), at all places v, the local 
Langlands parameters of II' (the Satake parameters when data are unramified) are bounded 
in absolute value by |- JV 2 +1 , where II' functorially lifts to a representation of GLn(A). This 
result follows from the local lift and the strong bounds of Luo, Rudnick and Sarnak [LRS99], 
and will be obtained once the details of the local lift are provided. 


We recall Theorem 2 of Kapl4c| : 

Theorem 4.12. Let r e Alg^pGL^ be unitary. Then r is distinguished if and only if 
L(s, r, Sym 2 ) has a pole at s - 0. 

We extend this result to square-integrable representations. Note that in particular, a 
supercuspidal distinguished representation must be self-dual. 

Theorem 4.13. Let r e Alg sqr GL n . Then r is distinguished if and only if L(s , r, Sym 2 ) has 
a pole at s - 0. 


Proof of Theorem \4.13[ Write r = {[vP-^Fp, z/W 1 )/ 2 /?])*, where p e Alg cusp GL fc is unitary 


and l = n/k. Assume that r is distinguished. First we have the following claim regarding the 
combinatorial structure of r. 


Claim 4.14. If t is distinguished, then p is self-dual and when l is odd, p is distinguished. 

The claim implies that r is self-dual, whence L(s,t x t) has a pole at s = 0. If n is odd, 
we immediately deduce L(0,r, Sym 2 ) = oo ( |Sha92j Corollary 8.2, L(0,r, A 2 ) < °° when n is 
odd). If n is even and l is odd, the claim and Theorem 14.121 imply L(0,p, Sym 2 ) = oo, hence 
according to the second case of Remark [4781 L(0,r, Sym 2 ) = oo. 

Assume n and l are even and suppose that r is distinguished with L(0,r, Sym 2 ) < 
Then L(0,p, A 2 ) < oo. The claim shows that p is self-dual, whence L(0,p, Sym 2 ) = 


oo. 


oo. 


Let G n be the split odd general spin group of rank n +1 (for details see |Asg02| IAS06, IHS121 
Kapl4a| ). In Kapl4a| we developed a theory of exceptional representations of G n , parallel 
to the small representations of S0 2n +i of Bump, Fricdbcrg and Ginzburg [BFG03, BFG06J (in 
the following argument one may take G n = S02n+i, the only loss is in a technical requirement, 
for the underlying local field to contain a 4-th root of unity). These are representations of a 
double cover G n of G n obtained by restricting the cover of Spin 2n+3 of Matsumoto [ Mat69 j. 
The definition of exceptional representations is similar to the definition for GL n , and they 
also have a simple parametrization. If 0 and 0' are a pair of exceptional representations 
of G n , an admissible representation k of G n , which admits a central character, is called 
distinguished if Hom Gn (@ ® 0', k v ) 4 0. 

In ||Kapl4c| we proved the following “inflation” result. Let P n be a maximal parabolic 
subgroup of G n with a Levi part isomorphic to GL„ x GLi. For n 0 e AlgGL n and sc C, 
define I(k 0) s ) = Indp"(5]/ 2 /q)| det | s ® 1). According to |Kapl4c| (Corollary 4.8), assuming 
that k 0 e Alg irr GL„ is tempered, I(k 0) 1/2) is distinguished for some pair 0 and 0' (more 
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specifically described in |Kapl4c| , but this will not be needed here) if and only if k 0 is 
distinguished. 

We claim that I(r, 1/2) is irreducible. Indeed according to Asgari, Cogdell and Shahidi 
[ACS 14 ] (Proposition 4.27), because L(0,p, Sym 2 ) = oo, I(p, 1/2) is reducible and for a real 
s 7 ±1/2, I(p, s) is irreducible. Then Theorem 9.1 (ii) of Tadic ([Tad98a], the proof is com¬ 
binatorial in nature and carries over from S02n+i to G n ) implies that I(r, 1/2) is irreducible 
(10,1/2) is is 1 / 2 d(p, l) » 1 in the notation of [ Tad98a] and l is even). 

Being irreducible, I(r, 1/2) is also generic. Now the assumption that r is distinguished 
implies that I(r, 1/2) is a quotient of a space 0® 0'. However, by Theorem 1 of |Kapl4c], as 
a representation of G n the space 0®0' does not afford a Whittaker functional, contradiction. 

In the opposite direction assume L(0,r, Sym 2 ) = oo. By virtue of Lemma 14.91 there exist 
a number field k with a ring of adeles A and a global cuspidal representation n of GL n (A), 
such that L s (l, n, Sym 2 ) = oo and at some place v, n,, = r. Bump and Ginzburg |BG92j 
proved that when L s (l, n, Sym 2 ) = oo, the following period integral is nonvanishing ( |BG92j 
Theorem 7.6) 

(4.11) f pu {g)(j>{g)<l>'{g)dg. 

J ZGL„(k)\GL n (A) 

Here Z is a subgroup of finite index in Cgl„(a) (Z = Cgl„(a) when n is odd); <pn is a cusp form 
in the space of n; 0 and <f>’ are automorphic forms in the spaces of two global exceptional 
representations of GL n (A) (defined in [ KP84 J Section II). The local-global principle implies 
that r is distinguished (see e.g. [JR92] Proposition 1). 


Proof of Claim \4CTf\ If l = 1, i.e., r is unitary supercuspidal, the result follows from Theo¬ 
rem 07[2J Assume l > 1. According to Zelevinsky |Zel80j (Proposition 9.6), 5 0 1|/2 (t v )u = 
rf 0 i/(!-0/ 2 p v with r i = ([v( 1 ~ l l/ 2 p, z/( z_3 )/ 2 p])L Then r v c rf x vC~ l )l 2 p v . Since r is distin¬ 
guished, 

Hom GLn ((0 n ,i )7 ® 9 ntl tf) Un _ k ' k ,6%l_ k ' k T? ®> // (1 -° /2 p v ) * 0, 

for some pair 7 and 7'. Hence by Lemma [4.21 for some 0 < j < min(n - k, k) = k, 

(4.12) Hom L „ fc . (Z ntkJ ® 1, (5^ . . ® 5 Q ] ,2 k _. {v^' 2 p) Ujk _. ) v ) * 0. 

Since p is supercuspidal, we must have j - 0 or k. 

If j = 0, the left-hand side of (14.12(1 becomes 


HoniGL n _ fe xGL fe 1,71 ® 7 ( 0 ) ^,1,72) ® 1,72) , H ® ^ ^ P )• 

Corollary 12.41 implies that this vanishes, unless vl l ~P! 2 p is distinguished. Since p is unitary, 
this is only possible when / = 1, but we are assuming l > 1. 

Therefore we must have j = k. Then <5g 1//2 fc (iq )u n ^k k ~ T 2 ®v( 1 ~ l ' > / 2 p, T 2 = zzh- 3 )/ 2 p])* ; 

and the left-hand side of (14. 12 p is nonzero only if both 


Hom GLn 2fc (fln-sfc.i.T! ® ^ n _2fc,i, 7 (, r 2 v ) * 0, Hom GLfc (p, p v ) * 0. 

The second condition implies p = p v . The first implies that r 2 is distinguished, and because 
t 2 e Alg sqr GL n _2fc, we can apply induction. If l is odd, the induction terminates with a 
supercuspidal representation of GL^, namely p, to which we apply Theorem 14.121 □ 

□ 
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Remark 4.15. Let r e Alg sqr GL n . If L(s,r, Sym 2 ) has a pole at s - 0, then I(r, s) (in the 
notation of the proof of Theorem \ is reducible at s - 1/2. Indeed, the theorem above 
implies that r is distinguished hence by [Ka pl4c| (Corollary f.8), I(r, 1/2) is distinguished. If 
I(r, 1/2) is irreducible, then it is also generic, contradicting the fact that ©<g>©' does not afford 
a Whittaker functional / |Kapl4c| Theorem 1). Of course this is not a new observation, as this 
reducibility already follows from the aforementioned result of Tadic 7[ Tad98a] Theorem 9.1). 

We are ready to prove the characterization result for generic distinguished representations. 


Theorem 4.16. Let r e Alg irr GL„ be generic and write r = t\k.. . .xr m , where Ti e Alg esqr GL^ 
and a = (ai,... ,a m ) is a partition of n. Then r is distinguished if and only if there is 
0 < m 0 < [m/2j such that, perhaps after permuting the indices of the inducing data, r 2 i = 
for 1 < i < m 0 and t* is distinguished for 2m 0 + 1 < i < m. 


Proof of Theorem, \4.16[ Since r is irreducible, one may permute the inducing data Ti®.. .<S>r m 
without affecting r. Hence one direction follows immediately from Corollary 14.71 

Assume r is distinguished. The result is trivial for m = 1, assume m > 1. We may also 
assume (perhaps after applying a permutation) that a,- < a m for all 1 < i < m. For each 
1 < i < m, set n = {[pi,v li ~ l pi]Y, U > 1, pi e Alg cusp GL a . /z .. Also put k = a m and l = l m . By 
Zelevinsky |Zel80] (Proposition 9.6), for any 0 < j < k , Sfy 2 .(r m )[/.,_. = 0 unless k/l divides 

'^cj,k—j 3i 3 

k - j and then 

= ([v a pmi V l ~ l pm]Y ® {[pm, ^ Pm])*, O = K k ~ j)/ k - 

Here 0 < a < l and since k/l divides k-j, a e Z. Note that if a = l {j = 0), {[u a p m , ix /_1 p m ])* = 1, 
and when a = 0 (j = k), ([p m , v a ~ x p m ]) 1 = 1. 

For 0 < j < n - k, write 

x ■ • • x = eij ® 02 J e Alg M n _ k _jj. 

Applying Lemma 14.21 for some 0 < j < min(n - k, k), 

(4.13) Horn Ln kij (f n ,k,j ® 1, (Pl,j ® Q 2 ,j ® ® (bmA^VmDT) * 

In particular 


H° m GL n _ fc _j xGL*._j ( ®7(j) 1,72 ) ® {^n-k-j, 1 , 7 ^ ®7^. ) @k-j, 1 , 7 ^ ) 5 

01 J®(([/°m > » / '"Vm]) t ) V ) *0 

and according to Corollary 12.41 pij and ([p m , ^ a_1 p m ]) i are distinguished. 

If j = 0, pi j = Ti x ... x r m _i is a distinguished generic representation and {[p m , iP , ~ 1 Pm\) t = 
r m . The proof is then complete, by applying induction to T\ x ... x r m _i. 

Henceforth assume j > 0 and then a < l. If also j < k, then since ([p m , zya_1 Pm]) i e 
Alg esqr GL fc _j is distinguished, Theorem 14. 131 implies {[p m , ^ a_1 p m ]) t = ([zy( 1_a )/ 2 p, z/G -1 )/ 2 ^]) 4 
for a self-dual p e Alg cusp GLPut A = (a - l)/2. Then p m = z/ _A p, 

s QlLi( Tm ) u i,k-i = 

and note that A + 1 < -A + / -1 and A > 0 (because 0 < a < l). When j = k, we cannot deduce 
anything about the structure of r m . 
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To proceed we need to describe the structure of gij ® g 2 j. To this end we use the concrete 
description of the composition factors of Sq ^ . ,(ri x ... x r m _\ )u n _ k _ :j , given by Zelevinsky 
( [Zel80] 1.6), which is a formulation of [BZ77] (Theorem 5.2) for this setting. Let B be 
the set of (m - 1) x 2 matrices whose coordinates are nonnegative integers, the sum of the 
coordinates in the i-th row is cq, the sum of the coordinates in the hrst column is n - k - j , 
and the second column sums up to j. Let b e B. The (i,r)-th coordinate of b will be denoted 
b{i, r ). For each 1 < i < m - 1, ( Ti)u b(i 1} b(i 2) = 0 unless cq/Zj divides b(i, 2) and then 




- 1/2 / \ 


= ([v ai pi,v li 1 p i ]) t ® {[pi,v ai 1 p i ]) t , o, = lib(i,2)/oti 


Note that if b(i,2) = a t , ([u ai pi,u li 1 Pi]) t = 1 and when b(i, 2) = 0, {[pi,v ai l pi\Y 
Assuming a,//,; divides b(i, 2) for each 1 < i < m- 1, put 

Qi,j,b = ([^ ai Pi, ^ Zl_1 Pi]) t x • ■ ■ x ([/"‘- 1 p m -iy lm - r V m -i]) i , 

02,j,b = ([pi, ^ 01 “Vl])* x ... X 


- 1. 


Then <5, 


- 1/2 


Qn 


in 


X Try 


171 1 ) U n _ k _j j 


is glued from the representations 


Ql,j,b ® Q2,j,bi b € B. 


The following claim uses this description to analyze the structure of P\. h b and Q 2 ,j,b, given 
that 02 .j,b contains a certain quotient. 


Claim 4.17. Let 7r e Alg cusp GL ^/i and assume that 0 < j < k, Ijlk e Z and for some b e B, 
(4.14) Rom GLj (g 2 , j! b,{['K,u l i- 1 n]) t ) 4 0. 

If j < k, then u^^pi = for some i with b(i, 2) > 0. If j = k, then for some 1 < i < m-1, 

Ti = ([ 7 T,v l ~ l 'K~]) t , and 


Ql,k,b = n X • • • X Ti_i X T i+ 1 X . . . X T m _ 1 . 

Now we can dispose of the nontrivial non-generic orbits. Indeed, assume j < k. Suppose 
(14.13ft does not vanish, then for some b e B, 

Hom GL .(p 2j - 6 , ([ u A ~ l+1 p , o~ A ~ ] p]) f ) * 0. 

Applying Claim [47171 we deduce that v ai ~ x pi = v~ A ~ l p for some i with b(i, 2) > 0. Then 

Ti = {[v~ A ~ ai p, i2- A - ai+li ~ 1 p]Y, r m = {[v~ A p, v-^-'p])*. 

We show that r* and r m must be linked, contradicting the irreducibility of r. Specihcally, 
we show 

— A _ ai <7 _ A, _ A — ai + k — 1 <c _ A + l — 1, —A — a h — 1 k — A — 1. 

Because pi e AlgGL a .^. and p e AlgGL fc /;, we have cq/Zj = Zc/Z, and since (by assumption) cq < 
k, we obtain l t < l. The hrst two inequalities follow from this and because a* = lib(i, 2)/cq > 0. 
The last holds because b(i, 2) < a* (by dehnition) whence ai < U. 

Finally, assume j = k. Then (j4.13j) vanishes unless for some b e B, both 

Hom GLji _ 2fc (0„_ 2 fc j i j71 ® 0 n -2k,l,Yii @l,k,b) * 0) Hom GLl (ft,/:,i)Tm) I 0. 

Again appealing to Claim 14.171 we find that 7 / = t ,^ for some 1 < i < m - 1 and 


01,fc,6 = ri x ... X Ti-i x r i+ i x ... x r m _ 1 . 

If n = 2k, we are done. Otherwise the theorem follows by applying induction to gi t k,b- 
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Proof of Claim \f.l r /\ Let 1 < R < ... < i c < m - 1 be a minimal set of indices such that 
b{i\. 2) + ... + b(i c , 2) = j, c > 1. Then 

Q2,j,b= ([PnX^Vu])* x x ([picX^'Vd)*- 

In general if e e AlgGL n is a subrepresentation of a representation parabolically induced 
from a cuspidal representation S\ ® ® e t e Alg cusp Mg, where (3 is a partition of n into t 

parts, we say that the cuspidal support of e is the multiset (i.e., including multiplicities) 
{a, and denote it by Supp(e). According to Zelevinsky [Zel80j (9.1 and 6.1) 

Supp( @ 2 ,j,b) = {Pir,--- ,^ air_1 Pi,}i<r<c Supp(([7r,z/fc“ 1 7r]) f ) = {n,.. . ,vH' -1 7r}. 

Assumption (14.1411 implies that ([7r, v fe' _1 7 r]) t is an irreducible subquotient of a representa¬ 
tion induced from the tensor of all the representations (with multiplicities) appearing in 
Supp(0 2 ,j,t>)- Hence by Bernstein and Zelevinsky |BZ77| (Theorem 2.9), as multisets 

lj 

(4.15) Supply,&) = {tt, ..., 

Therefore pi r = u Si rn for some integer Si r > 0, for all r. Furthermore, there exists i € 
{A, ••■Ac} such that u ai ~ l pi = and clearly b(i, 2) > 0. This completes the proof for 

3 < 

Henceforth assume j - k. If c = 1, then for i = i\ (1 < i < m - 1), a* > b(i,2 ) = k. Since 
fc > ay, we see that b(i,2 ) = ay, £> 2 ,fc,& = A and (14.14p implies a = ([tt, zF _1 7r])L Also because 
b(i , 1) + 5(i, 2) = ay and the sum of entries in the second column of b is k, we have b{i, 1) = 0, 
b(r, 2) = 0 and 6(r, 1) = ay for all r + i. Thus 

Ql,k,b = A X ... X Tj_i X Tj +1 X . . . X An-1, 

as claimed. 

Assume c > 1. For simplicity, assume z r = r for 1 < r < c. Now we can denote, for 1 < i < c, 

[pi,u li ~ 1 p i ] = [si,e i ], [pi,u ai ~ 1 p i ] = [s i ,h i ] 1 Si < hi < ei, s i ,h i ,e i ^'L. 

Indeed, the representations pi are all unramified twists of one representation 7r, so pi can 
be safely excluded from the notation. In other words, the segment {/y,..., v^^pf) will 
henceforth be denoted [a, a]. Note that a = {[Pi, ^ li ~ l Pi]Y = ([v Si 'K,i' ei Tr]) t . We can rewrite 
(I4.15P in the form 

(4.16) {zWtt, ..., u h C r}i<j< c = (tt, — , ^ _1 7r}. 

In particular 0 < Sj < / -I for all i. Renumbering if necessary, we can assume that the segment 
[si, ei] is such that a = 0 and e* < e± for all i such that s t = 0. 

Because pi e AlgGL a ./y and n € AlgGL fc ^, we have aii/li = k/l, and since (by assumption) 
oii < k, we obtain h < l. Since e t - sy + ! = /,;</ and a = 0, e± < l - 1. If s± < Si < e± + 1, then 
Si < ei < e\ because otherwise the segments [A,ei] and [a,a] are linked contradicting the 
fact that r is irreducible. In particular, Si + e i + 1 for all i. If e\ < l - 1, zA 1+1 7 r must appear 
in Supp(^ 2 , j,b), but this is impossible because e\ + 1 is not covered by any segment [a,a]- 
Therefore e\ = l - 1, then Sj < ei, whence a < e\ for all i. 

Thus far we have shown [a,a] = [0,Z — 1], which immediately implies a = ([ 71 , is l ~ 1 7T]) t , 
and all other segments are contained in the first segment. Next we argue that the segments 
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\si,hi\ are disjoint. Indeed, any such segment contributes at most /ij-Sj + 1 = a* = /*&(«, 2)/aij 
representations to Supp(p 2 j,f>) and since /j/ctj = l/k, 

Y j h i -s i + 1 = j;Y,b(i,2) = 1. 

i =1 i= 1 

But Supp(([7r,^ _1 7r]) t ) contains l non-isomorphic representations, hence by (I4.16P the seg¬ 
ments are disjoint. 

Therefore, perhaps after renumbering segments 2,... c, we can assume Si < s 2 ... < s c . 
Then the fact that their lengths sum up to l also implies Sj+i - hi + 1 whence 

Si < hi < s 2 < h 2 < s 3 ... < s c < h c . 

Because the segments \si,ef\, [sj + i,ej + i] are not linked and s* < s *+1 < e« + 1 (hi < ef), we 
obtain e c < ... < e 2 < e\ < l - 1. Now we must have e c = l - 1, because z/ z_1 7r e Supp(^ 2 j,fe) only 
if h c >l- 1. It follows that h c = e c and ei = ... = e c . 

We conclude [h, + 1, e,] = [s i+1 , e i+ i] for all 1 < i < c - 1 and 

([u hl+1 7T, z/ 61 ^])* X ... X ([z/ /lc ' 1+1 7T, Z^ ec_1 7r] )* = ([p S 2 7T, X ... X ([^ C 7T, Z/^Tr])* = T 2 X . . . X T c 

For the last segment [s c , h c ] = [s c , e c \, so b(c, 1) = 0 and 

Qi,j,b = ([v hl+1 7T, z/ ei 7r])* X ... X ([z/ fc «=- 1+1 7r, z/^-W])' X r c+ i X ... X r m _i = r 2 x ... x r m _i. 

Together with ri = ([zr, z/ z_1 7r]) t (proved above), the proof is complete also when j = k and c > 
1. Note that the indices were permuted during the argument, in the original representation 
Ti will actually be r* for some 1 < i < m - 1. □ 

□ 


Remark 4.18. The only place in the proof where we assumed that the characteristic of the 
field is 0, was in the application of Theorem \4.13[ However, it is simple to see that we only 
used the fact that a square-integrable distinguished representation is self-dual. Granted this, 
Theorem \f.lb\ is valid also for fields of odd characteristic. 


Corollary 4.19. Let r e Alg irr GL n be a generic distinguished representation. Then r is 
self-dual. 
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